(19) 



J) 



(12) 



(43) Date of publication: 

03.05.2000 Bulletin 2000/18 

(21) Application number: 99115494.9 

(22) Date of filing: 05.08.1999 



Europaisches Patentaml 
European Patent Office 
Office europeen des brevets (11) EP 0 998 124 A2 

EUROPEAN PATENT APPLICATION 

(51) lnt.Cl7: H04N 1/40, H04N 1/00 



(84) Designated Contracting States: 


(72) Inventors: 


AT BE CH CY DE DK ES R FR GB GR IE IT LI LU 


• Stelnle, Michael J. 


MG NL PT SB 


Fort Collins, CO 80525 (US) 


Designated Extension States: 


• Jofinson, Dan S. 


AL LT LV MK RO SI 


Greeley, CO 80634 (US) 


(30) Priority: 30.10.1998 US 183304 


(74) Representative: 


(71) Applicant: 


Schoppe, Fritz, Dipl.-lng. 


SchoppO} Zimmermann & Stdckeler 


Hewlett-Packard Company 


PatentanwSite 


Palo Alto, California 94304 (US) 


Postfach 71 08 67 




81458 Munchen (DE) 



(54) Optical inr^age scanner with internal measurement of point-spread function and 
compensation for optica! aberrations 

(57) An image scanner uses optical targets (120) 
within the scanner to characterize imperfections of a 
lens system (116) and to partially compensate for the 
imperfections using digital image processing. In one 
sanrple embodiment, a series of two-dimensional opti- 
cal targets are placed outside the document scanning 400^ 
area. Each individual target is suitable for obtaining an 
estimate of the point-spread function for a small seg- 
ment of the scan line. Each point-spread function is then 
used to compute a convolution kernel for the corre- 
sponding segment of the scan line. Alternatively, each 
point-spread function may be used in an iterative solu- 
tion for a modified image. In an alternative embodiment, 
a two-dimensional known random pattern is provided for 
a torget. Cross-conrelation of a portion of the known ran- 
dom pattern with the scanned image of the same por- 
tion of known random pattern provides an estimate of pig ^ 
the point-spread function for the portion of the scan line 
con-esponding to the portion of the known random pat- 
tern. Providing a series of targets or continuous random 
target over the width of the scan line, within the scanner, 
enables determination of the point-spread function as a 
function of position for an assembled lens, at the humid- 
ity and temperature appropriate for the scan. As a 
result a smaller, lower cost lens can be used and son^e 
the lens aberrations can be removed from the final 
scanned image. 
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Description 

FIELD OF INVENTION 

[0001] This invention relates generally to optical s 
systems used in image scanners, digital copiers, fac- 
simile machines, and similar devices and more specifi- 
cally to compensation of optical aberrations after 
measuring, within the scanner, in near real-time, the 
point-spread function of the optical system. io 

BACKGROUND OF THE INVENTION 

[0002] Image scanners convert a visible image on a 
document or photograph, or an image in a transparent is 
medium, into an electronic form suitable for copying, 
storing or processing by a computer. An image scanner 
may be a separate device or an image scanner may be 
a part of a copier, part of a facsimile machine, or part of 
a multipurpose device. Reflective image scanners typi- so 
cally have a controlled source of light, and light is 
reflected off the surface of a document, through an 
optics system, and onto an array of photosensitive 
devices. The photosensitive devices convert received 
light intensity into an electronic signal. Transparency 25 
image scanners pass light through a transparent image, 
for example a photographic positive slide, through an 
optics system, and then onto an array of photosensitive 
devices. 

[0003] In general, image scanners use an optical 30 
lens system or optical waveguide system to focus an 
image onto an array of photosensors. Lens systems and 
optical waveguide systems are subject to various optical 
aberrations such as spherical aben^tion. coma, astig- 
matism, field curvature, chromatic aberration, motion 35 
blur, and stray reflections (ghosting). Aberrations may 
be reduced by making the optical systems more precise 
and more complex, which in turn typically increases 
cost, size and weight. There is need for overall system 
cost reduction and size reduction by enabling use of rel- 40 
atively simple and low cost optical systems. 
[0004] Some known, fixed, aberrations may be 
rerrxjved by geometric calibration. For example, an 
Image of a grid of dots may be used to determine spatial 
pixel translations for images generated by fisheye 45 
lenses. Other compensation for some types of abenra- 
tions may be determined by using a n^thematical 
model of the lens and optical design software. 
[0005] Another method for compensation of optical 
aben'ation involves linear system theory and the so 
impulse response of the system. In optical systems, the 
impulse response, or the image of a bright point object 
against a dark background, is called the point-spread 
function. The two-dimensional Fourier transform of the \ 
spatial domain point-spread function is called the optical 55 
transfer function. The frequency domain optical transfer 
function includes magnitude and phase. A two dimen- 
sional, magnitude only, vertical slice through the optical 



transfer function is called the modulation transfer func- 
tion. If an image is blurred by an optical system, some 
blurring can be removed in the frequency domain by 
dividing the transform of the blurred image by the optical 
ti'ansfer function for the same imperfect optics. Convolu- 
tion in the spatial domain is equivalent to multiplication 
in the frequency domain. Accordingly, as an alternative, 
some blurring may be removed in the spatial domain by 
convolution with an appropriate two-dimensional kernel 
computed from the point-spread function. Because of 
aliasing, and because of possible desired trade-offs 
between signal-to-noise and image sharpening, the 
desired modulation transfer function may have a lower 
frequency response than an ideal modulation transfer 
function based on a response to a ti-ue impulse. 
[0006] Typically, determination of aberrations using 
a mathematical model of a lens, or determination of the 
point-spread function, is performed as a one-time cali- 
bration for on-axis imaging. However, the point-spread 
function of a lens may vary witii manufacturing proc- 
esses, may vary over time, may vary with temperature 
and humidity, and may vary with tine locations of the 
image focal plane and the object focal plane. If an opti- 
cal system is substantially different than a computer 
model, or if the lens system changes after determina- 
tion of the point-spread function, a resulting filtering 
operation to compensate for aberrations may further 
degrade the image instead of reducing innge degrada- 
tion. 

[0007] In particular, some low cost plastic lenses 
are relatively sensitive to temperature. It is common for 
an image scanner to include a lamp that radiates sub- 
stantial heat. Copiers and multi-purpose devices may 
also include heaters for fusing toner onto paper or for 
drying ink. Before scanning is requested, lanps and 
heaters may be off or in a low-power standby condition. 
During scanning, the internal temperature of the image 
scanner may change substantially over a period of time 
tiiat is long relative to the time required to scan a single 
image. In addition, for compact product size, image 
scanning lens systems typically focus on a document 
that is relatively close to the lens and therefore must be 
accurate over a wide field of view. Some aberrations, for 
example, astignnatism. coma, curvature of field are spa- 
tially variant. Therefore, a single kernel determined 
once for on-axis viewing may not be adequate for a lens 
system in an image scanner. 

[0008] There is a need within Image scanning prod- 
ucts for simple, low cost, rapid characterization of aber- 
rations, and compensation for aberrations, within a 
completed product. In addition, there is a need to com- 
pensate for aberrations, within scanning products, that 
vary spatially across tiie field of view of the lens. In addi- 
tion, there is a need to determine tiie required compen- 
sation before each scan to accommodate changes in 
tenperature. humidity, and other time varying factors. 
Finany. there is a need to monitor aberrations during a 
scan and to rescan if necessary. 
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SUMMARY OF THE INVENTION 

[0009] A goal of the present application is to char- 
acterize the imperfections of a lens system in an image 
scanner, and to make the measurements within the s 
scanner in near real time, and to partially compensate 
for the imperfections using digital image processing. In 
one example embodiment, a series of two-dimensional 
optical targets are placed outside the document scan- 
ning area. The targets extend over the entire length of io 
the scan line to enable compensation across the field of 
view of the lens system. In a variation, targets are also 
positioned alongside the document to enable monitor- 
ing of compensation during a scan. 
[0010] Each individual target is suitable for obtain- is 
ing an estimate of the optical transfer function for a 
small segment of the scan line. One example target has 
step functions in intensity (white-to-black or black-to- 
white) at multiple angles. Each step function is suitable 
for obtaining an estimate for one modulation transfer 20 
function. The aggregate of the multiple modulation 
transfer functions for one individual target is used to 
estimate the optical transfer function for the viewing 
angle corresponding to the spatial location of the indi- 
vidual target. 25 
[0011] In an alternative embodiment, a two<limen- 
sional known random pattern is provided for a target. 
Cross-correlation of a portion of the known random pat- 
tern with the scanned image of the same portion of 
known random pattern provides an estimate of the 30 
point-spread function for the portion of the scan line cor- 
responding to the portion of the known random pattern. 
[0012] Each indivklual optical transfer function is 
then used to conrpensate for optical aberrations. In one 
enfibodiment, the optical transfer function is used to 35 
compute a convolution kernel for the con-esponding 
segment of the scan line. The scanned image pixels 
within each segment of the scan line are convolved with 
a kernel appropriate for tiie segment In an alternative 
embodiment, an iterative solution is used to compute a 40 
compensated image. 

[001 3] Providing a series of targets, or a continuous 
random target over the width of the scan line, within the 
scanner, enables determination of the optical transfer 
function, as a function of position for an assembled lens, 4S 
at the humidity and temperature appropriate for the 
scan. Additional targets alongside a document enable 
ongoing verification during a scan. As a result, a 
smaller. lower cost lens can be used and some image 
degradation can be removed from the final scanned so 
image. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0014] ' 55 

Figure 1 is a cross-section side view of an example 
image scanner witii a calibration target in accord- 



ance with the invention. 

Figure 2 is a plan view of an example calibration tar- 
get. 

Figure 3 is a plan view of an alternative calibration 
target. 

Rgure 4 is a top view of the scanner of figure 1 illus- 
trating additional targets alongside a document. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENT OF THE INVENTION 

[0015] Figure 1 illustrates part of an example image 
scanner 100. The scanner 100 includes a transparent 
platen 1 02. A document 1 04 to be imaged is placed face 
down on tiie transparent platen. A light ray 106 origi- 
nates from a lamp 108, is reflected off the face of the 
document, and then is reflected off mirrors 110.112 and 
114. passes through a lens system 116 and finally onto 
a photosensor an-ay 1 18. Other configurations are com- 
mon. In general, lens system 116 may comprise multi- 
ple elements. In a scanner in accordance with the 
Invention, the optical performance of the lens system 
116 is characterized witiiin the scanner, and the scan- 
ner partially compensates for lens aberrations using 
digital image processing. Of particular interest is a lens 
calibration target 120. Target 120 is preferably placed so 
that calibration patterns on the target are at the same 
focal plane as the face of tiie document, and target 120 
is preferably placed so that it does not interfere with 
imaging the document 104. 

[001 6] Rgure 2 is an example embodiment of target 
1 20. Target 1 20 is preferably at least as wide as the wid- 
est document that can be scanned by scanner 100. In 
the example embodiment of figure 2, there are six iden- 
tical patterns 200. each of which is a semicircular array 
of alternating black and white radial wedges. The choice 
of the number six is for convenience of illustration only, 
and an actual target may have fewer than six patterns, 
or more than six patterns, depending on the require- 
ments of the lens system. 

[0017] For each pattern on target 120, because of 
aliasing, each black to white (and white to black) transi- 
tion in intensity should not be a true step function, but 
instead should be rapid, with a known rate of change, 
and ideally the modulation transfer function of the target 
shouW be known. Each intensity transition edge can be 
avenged along its radial extent to reduce noise. An 
edge-spread function can be computed for each edge. 
The resulting edge-spread function can be differenti- 
ated to produce a one-dimensional component of the 
point-spread function at one particular angle. See. for 
example. Kenneth R. Castleman. Digital Image 
Processing, Prentice Hall. 1996, chapter 16. If the 
point-spread function is circularly symmetrical then a 
one-dimensional point-spread function confuted from 
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a single edge can be rotated to produce the two-dimen- 
sional point-spread function. However, in general, 
edges at many orientations may be needed to accu- 
rately determine the two-dimensional point-spread func- 
tion. 

[0018] Rgure 3 illustrates an alternative target pat- 
tern. In figure 3, the targets are concentric rings. Con- 
centric rings enable measurement of an edge-spread 
function at any arbitrary angle, with multiple transitions 
to enable averaging to reduce noise. 
[001 9] Instead of a series of discrete calibration pat- 
terns as illustrated in figures 2 and 3, an alternative cal- 
ibration target can have a continuous random two 
dimensional pattern. Fa example, a suitable pattern in 
the spatial domain may be obtained by taking the two- 
dimensional inverse Fourier transform of a two-dimen- 
sional spectrum having constant amplitude and random 
phase. The cross-correlation of the known random pat- 
tern and the scanned image of the pattern is the point- 
spread function of the system. The random pattern can 
be logically divided into variable sized sections as 
needed to determine the point-spread function over var- 
ious segments of the scan line. 
[0020] When using the point-spread function for 
compensation, the lens system is assumed to be a lin- 
ear system. Given an original image f(xy) and the spa- 
tial domain point-spread function of the lens system 
h(x.y). the resulting scanned image g(x.y) is: 

where indicates two<iimensional convolution. The 
discrete form is as follows: 



9M^^Z^{x-m.yn)f{m,n) 

m n 



Given the scanned image g(x,y), and given h{x,y), then 
the original image is restored (blumng is reduced) by 
deconvolution: 

K^,y) = s{x,y) * g(x,y) 

where s(x.y) is a spatial domain kernel (deconvolution 
matrix, or inverse filter) computed from h(x,y), and per- 
haps nnodified to also incorporate a desired point- 
spread function. Given the system point-spread function 
h(u.v) in the frequency domain. s{x.y) may be computed 
by computing the inverse Fourier transform of 1/(h(u,v)). 
Alternatively, given the spatial domain point-spread 
function h(x,y), then s(x,y) may be computed by solving 
a series of simultaneous equations. 
[0021 ] Alternatively, instead of expressly computing 
s{x.y). f(x,y) may be computed directly by iteration. See. 
for example. Michael Elad and Arie Feuer, "Restoration 
f a Single Superresolution Image from Several Blurred. 



Noisy, and Undersampled Measured Images," IEEE 
Transactions on Image Processing, vol. 6, no. 12, pp 
1646 - 1658, December. 1997. Using a Steepest 
Descent algorithm, the following iterative equation may 
5 be used. 

<i^\ih \-hfj] 

where h is the spatial domain point-spread func- 

10 tion as defined above, f,o, the initialization vector can be 
any vector, and where \i>0 is smalt. 
[0022] One advantage of the iterative approach is 
that point-spread function compensation can be com- 
bined with two-dimensional resampling. Often, the 

15 desired scanned image may have a sample rate that is 
less than the highest possible optical sampling rate. 
High resolution color images require a lot of memory, 
and image analysis or transmission over a communica- 
tions channel may require a substantial amount of time. 

20 Accordingly, an operator may choose a lower sampling 
rate to reduce memory, processing time, or transmis- 
sion time. An image may need to be scaled to fit within 
a frame in a document or display, with a sampling rate 
that is determined by the resolution of a printer or dis- 

25 play device. As a result, the desired sampling rate may 
be any arbitrary number. Rnally, in some scanning con- 
figurations, scanning time may be reduced if sampling 
rate is reduced. Scanning time is often limited by an 
exposure time per pixel which is required to obtain a 

30 particular signal-to-noise ratio. Charges for adjacent 
CCD elements nnay be combined to enable a faster, but 
lower sampling rate, scan. 

[0023] For any of the above reasons, scanner oper- 
ators often request some fraction of the highest optical 

35 sampling rate. Typically, sanrtples for a single scan line 
are read by the scanner at the highest optical sampling 
rate, intermediate interpolated samples are computed, 
and the desired sampling rate is obtained by resampling 
the interpolated samples. For example, a scanner may 

40 have a 1,200 pixel per inch optical sampling rate, and 
may interpolate to 4,800 pixels per inch, and then resa- 
mple the 4,800 pixels per inch samples to provide an 
image having 938 pixels per inch (where 938 is just an 
arbitrary example for illustration). For scanners having a 

45 moving linear sensor array, interpolation and resam- 
pling may provide an arbitrary resolution along the 
scanline, and resolution in the direction of scanning may 
be controlled by the translation speed of the sensor 
array. Alternatively, bilinear interpolation may be used to 

so compute a pixel value using the intensities of the four 
dosest neighboring sensor values. However, as dis- 
cussed in the article by Elad referenced above, com- 
pensation for blurring (by using the point-spread 
function) and two-dimensional resampling may be com- 

55 bined in the iterative approach. 

[0024] While there are advantages to the iterative 
approach as discussed above, many scanners have 
specialized hardware support for spatial domain kernel 
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Operations. Accordingly, in the following discussion, the 
kernel approach is used for illustration. In general, the 
point-spread function is only part of the information 
used to compute the kernel. 

[0025] Computation of kernels from edge-spread 5 
data may be performed within the scanner. Alternatively, 
for peripheral devices, edge-spread data may be 
uploaded to a host computer. The host computer may 
then compute the kernels and download the kernels to 
the scanner. An image scanner may also need to con- io 
volve an image with other kernels, for example, for 
image sharpening. The various kernels can be con- 
volved together, and the resulting comt)ined kernels can 
be applied to the scanned image data. Some kernel 
information may be determined by a host computer. For is 
device independence, me scanner may need to upload 
internal scanner kernel information to the host compu- 
ter. The host computer may then convolve scanner spe- 
cific kernels with kernels determined by the host 
computer and download the resulting combined kernels so 
to the scanner. See. for example. U.S. Patent applica- 
tion number 08/775,061 . In accordance with the present 
invention, the scanner may upload edge-spread data to 
the host computer in addition to other kernel informa- 
tion. 25 
[0026] With N patterns, the scan line is divided in N 
regions, and the point-spread function is assumed to be 
constant within each region. For each region, a convolu- 
tion kernel is computed from tiie corresponding point- 
spread function and desired modulation transfer tunc- 30 
tion. The kernel size is preferably greater than the extent 
of any blunring. For example, a kernel of nine-by-nine 
values may be used if blurring of one pixel does not 
extend beyond 4 pixels in any direction. For each 
region, In the spatial domain, a convolution kernel is 35 
used to operate on each pixel within the region. Pixels 
near the document edges n^y ren^ain unfiltered, or 
edges may be extended using null data to enable filter- 
ing to extend to the edges. If optics compensation ker- 
nels are convolved with other kernels, then the optics 40 
compensation kernel that is convolved with other ker- 
nels will be dependent on the region being scanned. 
[0027] Figure 4 is a top view of a scanner illustrating 
additional calibration patterns alongside a document 
Figure 4 shows patterns 400 on only one side of the 45 
document, but they may also be placed on both sides of 
the document. If temperature change is relatively rapid 
during a scan, the point-spread functions computed 
before the scan may become inappropriate. Additional 
patterns alongside a document enable checking to see so 
if at least one of the point-spread functions is changing 
during the scan. For purposes of monitoring for change, 
it may not be necessary to compute an entire point 
spread function. It may be sufficent, for example, to 
monitor one edge-spread function at one angle. If the ss 
measured function is changing in real time, and if the 
lens system changes consistently, monitoring patterns 
as in figure 4 may provide sufficient information to per- 



mit real time modification of the point-spread functions. 
Alternatively, the scanner may simply go back to the 
beginning, ar»d recalibrate and rescan. 
[0028] incorporating calibration targets within a 
scanner also facilitates compensation for operater con- 
trolled variables, including resolution and motion blur. If 
there is continuous relative movement between a sen- 
sor array and a document being scanned, some blur 
results from movement during the exposure time for 
each pixel. If the amount of motion is known, then 
motion blur can be incorporated into the compensation 
kernel. See. for example, the article by Elad referenced 
above. In many scanners, the optical sampling rate in a 
direction perpendicular to a scan line Is determined by 
the speed of the relative scanning motion. For example, 
if an operator asks fa a high resolution, the relative 
scanning motion is relatively slow, and if an operator 
asks for a low resolution, the relative scanning motion is 
relatively fast. With calibration patterns internal to the 
scanner (as in figures 2 and 3), the patterns should be 
scanned at the (X,Y) resolution specifed by the opera- 
tor. Then, image specific resolution and motion blur 
become inherent to the calibration process. 
[0029] In summary, providing multiple calibration 
target patterns within a scanner enables use of a lower 
cost lens system, where the lens system may have opti- 
cal cfwacteristics that may vary from lens to lens and 
may vary with temperature and time. In addition, provid- 
ing calibration target patterns within a scanner facili- 
tates computation of image specific compensation (for 
exanple. specific (X.Y) resolution and motion blur). In 
addition, no ROM is required to store extensive tables of 
predetermined point-spread data. 
[0030] The foregoing desaiption of the present 
invention has been presented for purposes of illustration 
and description. It is not intended to be exhaustive or to 
limit the invention to the precise form disclosed, and 
other modifications and variations may be possible in 
light of the atxjve teachings. The embodiment was cho- 
sen and described in order to best explain the principles 
of the inverttion and its practical applk:ation to tiiereby 
enable others skilled in the art to best utilize the inven- 
tion in various embodiments and various modifications 
as are suited to the particular use contemplated. It is 
intended that the appended claims be construed to 
include other alternative embodiments of the invention 
except insofar as limited by the prior art 

Claims 

1, A method of scanning an image using an image 
scanner, the method comprising the following 
steps: 

scanning, through an optical system (116), an 
optical calibration pattern (120) within the 
scanner; 

computing a point-spread function from the 
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scanned optical calibration pattern; 
scanning an inoage (104) to obtain image pix- 
els: and 

using the point-spread function to modify the 
image pixels to remove some aberrations s 
caused by the optical system. 

2. The method of claim 1 , further conprlsing: 

the step of computing a point-spread function io 
further comprising: 

computing a spatial domain digital kernel 
from the point spread function; and 

15 

the step of using the point-spread function fur- 
ther comprising: 

covoMng the image pixels with the spatial 
donriain digital kernel. 20 

3. The method of claim 2, further comprising: 

computing multiple different digital kernels, 
each kernel corresponding to a portion of a 25 
scan line; and 

convolving each digital kernel with image pixels 
falling within the portion of the scan line corre- 
sponding to the kernel. 

30 

4. The method of claim 1 . the step of using the point- 
spread function further comprising: 

iterating an equation that modifies the image 
pixels to form a new image with some aberra- 35 
tions, caused by the optical system, removed. 

5. An image scanner comprising: 

an optical system (1 1 6) : and ^ 
at least one optical target (120). the optical tar- 
get suitable for determining an optical transfer 
function for the optical system. 

6. The image scanner of claim 5. further comprising: 45 

at least one additional optical target (400). the 
one additional optical target positioned in a 
location so that the one additional optical target 
may be used to verify the optical transfer func- so 
tion determined in claim 5. during a scan. 
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(54) Optical Image scanner with internal measurement of point-spread function and 
compensation for optical aberrations 



(57) An image scanner uses optical targets (120) 
within the scanner to characterize imperfections of a 
lens system (116) and to partjally compensate for the 
imperfections using digital image processing. In one 
sample embodiment, a series of two-dimensional opti- 
cal targets are placed outside the document scanning 
area. Each individual target is suitable for obtaining an 
estimate of the point-spread function for a small seg- 
ment of the scan line. Each point-spread function is then 
used to compute a convolution kernel for the corre- 
'sponding segment of the scan line. Alternatively, each 
point-spread function may be used in an iterative solu- 
tion for a modified image. In an alternative embodiment, 
a two-dimensional known random pattern is provided for 
a target. Cross-correlation of a portion of the known ran- 
dom pattern with the scanned image of the same por- 
tion of known random pattern provides an estimate of 
the point-spread function for the portion of the scan line 
corresponding to the portion of the known random pat- 
tern. Providing a series of targets or continuous random 
target over the width of the scan line, within the scanner, 
enables determination of the point-spread function as'a 
function of position for an assembled lens, at the humid- 
ity and temperature appropriate for th scan. As a 
r suit, a smaller, lower cost lens can be us d and some 
the lens aberrations can be removed from th final 
scanned image. 
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